Perceptually Adaptive Graphics

The ultimate goal of realistic image synthesis is to generate computer imagery that is perceptually indistinguishable from its real-world counterpart. Many rendering techniques have been proposed, making tremendous progress to reach this goal. Now we are faced with two basic questions. How far we are from the goal? How well the existing techniques perform? No doubt a quantitative definition or error metric of image realism is important for systematically evaluating synthesized images and comparing different techniques. However, such a definition or metric has not been available. This problem will remain open in the near future considering that the process of human evaluation of image realism, which involves complicated physical as well as physiological factors, has not been understood sufficiently so far. Nevertheless, we can adopt another approach that does not rely on the entire understanding of the human evaluation process. The solution is based upon perceptual limits, the maximal capabilities of the human visual system. It is known that the basic aspects of realism include space, color, and motion. As beginning one can focus on one aspect and establish a simple metric based upon the reference of the perceptual limit. This metric can be improved gradually within the aspect by including more factors and considering the application feedback, and be eventually incorporated with other aspects to form a comprehensive metric. Here we present an example of this approach: a CIELab-based error metric for the color aspect. Perceptual Limits The perceptual limit for space, which is also called the minimal resolvable spatial distance, is about 0.084 mm. This value can be calculated by applying Rayleigh's criterion for the eye (cf. page 463 in [Hechtt98]) to a viewing distance of 25 cm, for which the eye has the best acuity. For color, the minimal resolvable color distance is about 1 unit in CIELab color space [Hunt95]. Although the color appearance model [Fairchild98] measures colors more accurately, as beginning the CIELab space is good enough. Finally, for motion, the critical flicker frequency is about 60 Hz (page 18 in [Glassner95]). The critical flicker frequency has been widely adopted as the basis for motion realism or smoothness in computer animation, but it appears that the minimal resolvable distances for space and color have not been considered as references in realistic image synthesis. Thus, one may regard perfect realism as the point where a synthesized image has just overreached the perceptual limits. At this point, the difference between a synthesized image and its real-world counterpart is beyond the maximal recognizing powers of the eye. Putting it in another way, further improving the rendering accuracy beyond the perceptual limits will not be differentiated by the eye and is therefore unnecessary. For this reason, it is reasonable that we base on the perceptual limits to establish error metrics for image evaluation. CIELab-Based Error Metric We propose a simple error metric based on color differences measured in CIELab space. When comparing two images (of the same size), we compute the CIELab color distances between all corresponding pixel pairs. Thus, given a CIELab distance value, we know how many pixel pairs are differed by this color distance. This forms a histogram, where the horizontal and vertical axes are respectively CIELab distance and the corresponding percentage of pixel pairs. The reference image is best acquired from measurement without losing any color information, such as a spectral image. It is also possible to use a synthesized image with very high accuracy as the reference image. If a histogram distributes mainly on small color distances, say below 3 units, the two images have a good correspondence. In particular, when a histogram distributes completely below 1 unit, the two images have reached a perfect correspondence. Note, however, that this error metric has not considered all relevant factors for image realism. For example, pixel-pixel correlations such as object boundaries and color contrast are important. This means that this metric can be improved by including contributions from these factors. The error metric can be applied in two ways in computer graphics. First, one can use Perceptually Adaptive Graphics 9 CAMPFIRES: Synergies for the Future ACM SIGGRAPH and EUROGRAPHICS it to evaluate the impacts of different parameter values within the same technique. Second, it can be used to compare different techniques. In both cases, the spatial parameters of the camera and objects should be fixed. Thus, the corresponding pixels in the two compared images correspond to the same spatial point, and their color deviation is caused only by using different parameter values or methods. Example: Comparing Composite Model and Sampling We apply the CIELab-based error metric to comparing the composite model [Sun00] and the sampling method on representing spectral functions. The composite model decomposes any spectral function into a smooth background and a collection of spikes. A spike is thus represented with a delta function, and the smooth background with uniform sampling. The decomposition makes the composite model much more compact than the sampling method to achieve the same accuracy. This advantage is particularly obvious for fluorescent spectra that have narrow but strong peaks. Our test scene contains six opaque spheres with reflectances "white", "red", "orange", "yellow", "green" and "blue" of the Macbeth ColorChecker [Wyczecki82]. The light source is a Mercury Arc lamp, which is fluorescent. Figure 1 displays the images rendered with raytracing using the sampling method of 1000 points (used as the reference image), the composite model with 12 sampling points for the smooth background, and the sampling method of 64 and 32 points, respectively. The CIELab distance histograms are displayed in Figure 2. When using the sampling method, even with sample points as many as 64, the error is still noticeable and the histogram has considerable distribution on large color distances. The reason is that the spikes are narrow and missing sampling a spike will result in a significant error. In contrast, the composite model using only 12 sample points has much better accuracy. All such differences are quantitatively shown by the CIELab-based histograms in Figure 2. Future Work Improve the CIELab-based error metric by including pixel-neighborhood factors such as object boundaries and color contrast. Measure color distances with color appearance model. If possible, map a color distance histogram to a single value. This can be done by integrate the histogram with a weighting function over color distance. The question is to find a reasonable eight function. Based on the color distances between the corresponding pixel pairs, we can also show the mage regions with large color disparities. This information is useful for analysis. Establish a spatial error metric based upon the minimal resolvable spatial distance. The perceptual limit for space can be used as a criterion for multi-resolution rendering. Combine the color aspect with space and motion to form a comprehensive error metric. Summary Statements A formal definition of image realism founded on the understanding of the human evaluation process will remain a open problem in the near future. This is because the human evaluation process is very complicated and has not been understood sufficiently. As a practical approach, one can establish an error metric for image evaluation based upon the perceptual limits. This approach does not rely on the entire understanding of the human evaluation process. As beginning, one can focus on one realism aspect and propose a simple error metric. The metric can be improved gradually within the aspect by including more factors and considering the application feedback, and eventually incorporated with other aspects to form a comprehensive metric. As the perceptual limits, the minimal resolvable spatial distance is about 0.084 mm, the minimal resolvable color distance is about 1 unit in CIELab color space, and the critical flicker frequency for motion is about 60 Hz. Perceptually Adaptive Graphics 10 CAMPFIRES: Synergies for the Future ACM SIGGRAPH and EUROGRAPHICS Figure 1: Rendered images using the composite model and the sampling method. The light source is a mercury arc lamp. From left to right, the images are rendered using the sampling method with 1000 points( used as the reference image), the composite model with 12 sample points for the smooth background, and the sampling method with 64 and 32 points, respectively. Compared to the first image, the gray and blue spheres in the fourth image have obvious errors and the red sphere in the third image is too dark. But the second image matches the first very well. Figure 2: Comparing errors of test images rendered with the composite model and sampling method. The illumination source is Mercury Arc (spiky SPD). One may regard perfect realism as the point where a synthesized image has just overreached the perceptual limits. A CIELab-based error metric is a histogram, where the horizontal and vertical axes are respectively CIELab distance and the corresponding percentage of pixel pairs of two images. If a histogram distributes mainly on small color distances, the images have a good correspondence. The CIELab-based error metric can be used to compare different parameter values as well as different rendering techniques. As an example, the CIELab-based error metric is applied to comparing the composite model and the sampling method on representing spectral functions. References: Hecht98 E. Hecht, Optics, Third Edition, Addison-Wesley, Reading, Massachusetts, 1998. Hunt95 R.W.G. Hunt, The Reproduction of Color, Fifth Edition, Fountain Press, England, 1995. Fairchild98 M. D. Fairchild, Color Appearance Models, Addison-Wesley, Reading, MA, 1998. Glassner95 A. S. Glassner, Principles of Digital Image Synthesis, Morgan Kaufmann, San Francisco, 1995. Sun00 Y. Sun, F. D. Fracchia, and M. S. Drew, "A Physically-Based Dual Representation of Spectral Functions," Optical Engineering, Vol. 39, No. 11, Nov. 2000, pp. 2931-2942. Wyszecki67 G. Wyszecki and W. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulas, Second Edition, Wiley, New York, 1982. © Copyright is held by the author, Yinlong Sun, 2001 Perceptually Adaptive Graphics 11 CAMPFIRES: Synergies for the Future ACM SIGGRAPH and EUROGRAPHICS Make Pictures, Not Images Jack Tumblin, Program of Computer Graphics, Cornell University All great pictures, no matter how they are made, recreate some desirable set of visual sensations and perceptions. Picture-making processes are human attempts to capture and control what we see, or what we would like to see. Accordingly, we should regard pictures as containers for perceived quantities rather than dismissing them as a simple set of recorded of light measurements. However, machine-assisted picture-making processes such as photography and computer graphics usually make just one narrow, restrictive class of pictures known as images. Our machines copy scaled 2D scene intensities to a display device--most other forms of picturemaking such as sketching, painting, or carving are more directly governed by visual appearance instead, and are often much more expressive and successful. I think we should move beyond digital image-making to develop new perceptuallyguided digital picture-making processes instead.